Repository navigation
test(verifier): check every canonical module once instead of 55 hand-kept lists - #13
Merged
Merged
Conversation
…iliation Six engine tests have failed on main since 22 October, when two changes landed without the tests that depend on them being updated: - Interface 48 (ad56443). Four publication tests asserted the literal "interface 47;" in the server instructions, and two contract examples still carried interface_version 47. The assertion now reads the number from the interface the server under test serves, so it cannot go stale again, and the examples say 48. - Publication checkout reconciliation (8c5a4ad). The create-and-retire apply test still asserted that apply leaves the working tree untouched. It now asserts the documented behaviour: HEAD, index and worktree present the published candidate and no other local difference exists. No production code changes. The suite-input manifest is regenerated; it was already stale on main because 5bce311 added files without refreshing it. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The MCP server opens the snapshot store for each tool call, and every open of an existing store ran PRAGMA integrity_check and foreign_key_check over the whole file. On a 282 MB store that is about 0.6 s added to every route, read and related call, growing with the file. The full audit now runs at most once per process for each store file, identified by device and inode. It still runs on the first open in every process, throughout a version 1 migration, and for a file put in place under a different inode. Application id, schema version and exact schema checks still run on every open. Measured on a copy of that store, five consecutive opens in one process go from about 0.6 s each to 0.6 s for the first and under 1 ms for the rest. Known limits, documented in the snapshots README: a one-shot CLI call is a new process and still audits; a store deleted and recreated in place can reuse its inode on ext4 and is then not audited again by a server that is already running. Tests cover one audit for two opens, a replaced file, a corrupt replacement, the checks that stay on every open, a fresh process rejecting a corrupted file, migration keeping all of its audits, and the server auditing once across several tool calls. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The recorder exists to show how agents use the tool interface so the
interface can be improved. As built it could not do that: it was off
unless two flags were passed, so the application purpose was never
recorded; it gave up for the life of the process after a 20 ms lock wait,
so servers started together recorded nothing; and it stored argument
shapes rather than values, so it could not say which standard was read,
which facts were sent or which rejection came back.
Always on. Every server process records to
${XDG_DATA_HOME:-~/.local/share}/standards-engine/observations/
interactions-v2.sqlite3 with no flags. --no-observe or
STANDARDS_ENGINE_OBSERVE=off turns it off. In-process MCPServer
construction records only when given a configuration, and every test and
harness launch passes --no-observe; a guard test fails if a new launch
site forgets.
Invisible. Tool results are unchanged, a healthy recorder writes nothing
to stderr, and a recording failure never changes or fails a call.
No dropped records. A busy database queues the row in a bounded in-memory
backlog and delivers it at the next call or at close. The database uses
write-ahead mode. Eight processes started at the same instant on a new
database now keep all 176 starts and terminals; the previous code kept 42.
Loss that does occur (backlog overflow) is counted in the database.
Seven-day window. Unchanged. A full file now evicts its oldest records
instead of refusing new ones. Schema v2 uses a new file; a v1 file is
refused and left untouched.
Useful values. Navigation calls keep targets, options and route facts as
sent, bounded to 200 characters per string. Authoring calls keep only
identifiers and enum words, never bodies, titles, rationales or evidence.
Results keep kind, status, code and the selected or returned module IDs.
A client thread id is stored only as a keyed hash, with a main or
subagent marker. No other request metadata is read.
The report gains a readable text summary (rejection rates, latency and
size percentiles, refused fact names and values, most-read targets,
repeat reads per thread) and a --since-days filter.
MCP-OBSERVABILITY.md is rewritten for the new contract.
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…labelled Application approval was a gate: one module without a current approval anywhere in a selection rejected the whole route with "awaiting application publication", naming nothing. 40 of 85 registered materials are not currently approved, including two of three application profiles, and any edit to Core or the Router would have taken every application call offline until re-approval. In recorded use, 43 of 91 application routes were rejected this way and agents responded by deleting true facts from their queries. Approval is now a label. This is interface 49. - Every registered material a call selects or names is delivered. Each read result and each reading-plan entry carries review_state: current, changed-since-review or not-reviewed. The approval manifest and the authoring-side exposure workflow are unchanged. - An unregistered or misspelt identity is rejected as APPLICATION.UNKNOWN_TARGET, and a traversal a relationship group does not permit as APPLICATION.TRAVERSAL_NOT_PERMITTED. APPLICATION.CONTENT_UNAVAILABLE is removed. - Application route content pages are sized for inline display: about 40 KiB of structured result by default (--route-page-bytes, 8 KiB to 2 MiB), always at least one whole module. The previous 2 MiB bound produced pages the client diverted to a file. Authoring paging is unchanged. - The application connection instructions and tool descriptions are rewritten in plain language: what the server is for, the sequence of calls, snapshot reuse and what the labels mean. A new application skill, .agents/skills/coding-standards/SKILL.md, teaches the same sequence with a worked example taken from real engine output. - Tests that hard-coded the interface number now read it from the installed interface. The generated contract and tool catalog are regenerated by the repository generator. The CI path filter gains the new skill folder. One current guide, docs/guides/positive-guidance-and-provenance.md, still describes the gate. It is pinned by a coverage certificate and is left for a certificate retirement. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…anged Every server start validated the contract schema against the JSON Schema metaschema: about 0.36 s of a roughly one-second start, for a file that is the same on every start. Real agent sessions paid it at every connection and the test suite paid it hundreds of times. The generator now records the digest of the schema it validated in the generated contract (VALIDATED_SCHEMA_DIGEST). The engine passes that digest to compile_contracts, which skips only the metaschema validation when the schema it was given has the same digest. Every other check still runs on every call. - A schema edited without regenerating has a different digest and is fully validated, as before, so an invalid or drifted schema is still rejected with CONTRACT.INVALID_SCHEMA. - The comparison is made inside compile_contracts against the schema in hand, rather than trusting a caller's flag. - The digest is of the JSON value, so key order and formatting do not matter, but 1, 1.0 and true are distinct, and a value that does not read back equal (a tuple, a non-string key) has no digest and is validated. - The generator, its --check mode and the contract_projection verifier check never pass a digest and always validate. Startup therefore relies on the validation done at generation time. A server answering only initialize goes from about 0.75 s to about 0.42 s on the development machine. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…ll cores The Engine suite took about 28 minutes serially on the development machine and about 44 minutes in CI. Three things made it slow: tests started a new server process for every call, two scenarios were run twice, and the suite ran on one core. One server process. A shared helper, tests/stdio_server.py, starts the real server once and exchanges newline-delimited JSON-RPC with it for as many calls as a test makes. It checks stderr after every response and requires a clean exit at close. The per-call launch helpers now use it. A test starts a further process only where its claim is about a separate process (a cold read after publication, completion or recovery in a fresh process, a continuation honoured by a process that did not issue it, two processes contending for the store, a launch-option check, the recorder's simultaneous-start tests); each such place carries a comment naming the claim. Server processes per suite run fall from 262 to 117. No scenario twice. BatchedPublicationTests no longer inherits and reruns the two publication walkthroughs; one focused test covers what was particular to it (compact summaries, the inline first work page, paging through workflow_details, mixed resolve_many batches, review reaching ready). The test that reran the whole supporting-workflow scenario on warm caches is merged into that scenario, which now runs once on per-purpose caches and asserts both were hit. Two things are no longer exercised: verify, apply and recover over MCP after a batch-completed review (batch and serial decisions producing the same authority is proved in process by test_review_workflow_ux), and the supporting-workflow scenario on cache-less facades (cache-less publication remains covered by test_agent_workflow, test_publication_recovery, test_analysis and test_navigation_indexes). All cores. tests/parallel_unittest.py runs one `python -m unittest` worker per test module, as many at a time as there are cores. unittest still discovers and runs every test. The run fails if a worker fails, or if the workers together ran a different number of tests than one serial discovery finds. CI uses it for the Engine suite. Running in parallel exposed tests that created a snapshot store inside the repository under test; they now use a temporary installation root. Measured on the development machine: serial 1,667 s before, 1,379 s after; 283 s on all 24 cores; 448 s with --jobs 4. Test count 670 (two reruns and one merged test removed, one focused test and four helper tests added). Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…kept lists Fifty-five metadata_graph checks each validated a hand-written list of module paths. The lists named 57 of the 74 canonical modules between them, repeated the same validation up to 55 times, and could not fail for a reason the others would not also report. A module added to the corpus was covered only if someone remembered to add it to a list. Replace them with one check that reads the canonical corpus manifest and validates every module it declares: - metadata_graph gains a `corpus` mode naming the manifest. - standards_metadata exposes canonical_module_corpus_members so the check and load_canonical_module_corpus share one reader. - New suite canonical-corpus-metadata holds the single whole-corpus check; a verifier test fails if the repository registers more or fewer than one. The removals were made by the Engine's maintain_evidence operation (preview, then apply), not by hand: 55 checks retired, the 50 suites left empty and their registry entries removed, five suite descriptions updated. No retained suite required a removed one, and the policy-impact catalog, relationships and coverage requirements are unchanged. Kept on purpose: release-maintenance-policy/metadata-closure. That suite is a node in the policy-impact catalog, so retiring it changes every coverage requirement ID and turns the 16 current certificates stale. It needs an owner decision. Suites 73 -> 24, checks 121 -> 67. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes
Fifty-five verification checks each validated a hand-written list of module paths. This replaces them with one check that validates every module in the canonical corpus.
metadata_graphchecksWhy the 55 were removed rather than fixed
What replaces them
metadata_graphgains acorpusmode that names the corpus manifest and validates every module it declares.canonical-corpus-metadata, holds the single whole-corpus check. A verifier test fails if the repository registers more or fewer than one.standards_metadataexposescanonical_module_corpus_members, so the check andload_canonical_module_corpusshare one reader.How the removals were made
The Engine's
maintain_evidenceoperation made them (preview, then apply), in a scratch clone. Nothing was deleted by hand. It retired the 55 checks, removed the 50 suites left empty and their registry entries, and updated five suite descriptions.The review evidence supplied to that call was the audit document, which is not committed in this pull request.
What was checked before removing
One check deliberately kept
release-maintenance-policy/metadata-closureis the same kind of path list, but its suite is a node in the policy-impact catalog. Retiring it changes every coverage requirement ID, which turns all 16 current certificates stale. That needs an owner decision, so it stays for now.Verification
verify.py --complete: 24 of 24 suites pass.Stacked on #12.
🤖 Generated with Claude Code